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Abstract 

City University of Seattle, a comprehensive teaching university that focuses on the needs of 
working adult students, sought to streamline its faculty recruitment and hiring practices by 
identifying a list of candidate screening criteria that could be used to identify adjunct faculty 
candidates who are likely to be successful as instructors. To this end, student evaluations and 
ratings by primary supervisors were used to separate faculty members into groups based on 
performance metrics. High performers were compared to the lower performers and differences 
between the groups were noted regarding factors such as education, professional and academic 
experience, and scholarly activity. The study revealed limited differences between the groups 
and calls into question some of the current assumptions regarding the typical minimum 
qualifications of adjunct business faculty. 



Introduction 

Accreditation agencies and others place a great deal of importance on business faculty hiring 
qualifications and typically emphasize such items as business PhDs or related doctoral degrees, 
evidence of scholarship, and teaching experience (Sixl-Daniell, Williams, & Wong, 2006; 

Swartz, Swartz, & Liang, 2007). In many cases, decisions are made regarding the likely success 
of a faculty candidate based solely on how well the candidate has represented his or her 
qualifications on a CV. Such a screening process and its related criteria may fit well for more 
traditional, research-based universities but is it an adequate model for the selection and hiring of 
faculty in a teaching university that is more likely to use adjunct practitioner faculty? Is it 
appropriate in an environment where more emphasis is placed on practical experience and 
teaching skill than on academic preparation and scholarship? Should all universities seek to 
fulfill their faculty positions with candidates that fit the traditional profile or is a different type of 
faculty member needed for different schools? And if there is a different profile for teaching 
universities, what factors should administrators look for in the initial screening process? 

This study sought to identify a correlation between qualifications used to screen adjunct business 
faculty and faculty performance by examining a group of business instructors at a teaching 
university. Several factors were examined including professional experience, academic 
experience, degree levels, and scholarship. These factors were compared to two primary 
measures of faculty performance: 1) student ratings of the effectiveness of instruction on end of 
course evaluations (EOCEs), often the subject of debates regarding their use and application, and 
2) a performance assessment received by the faculty member’s primary supervisor. The goal of 
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these comparisons was to discover if there was a reliable link between the initial screening 
criteria and specified measures of performance, related to instruction, in a university that utilized 
a practitioner faculty model. 



Background 

University business programs, primarily those at research institutions, are facing increasing 
difficulty in their attempts to find faculty that fit accreditation standards (Swartz, Swartz, & 
Liang, 2007). In many cases, faculty positions remain unfilled or are filled on a temporary basis 
by part-time faculty because full-time qualified faculty cannot be found. As Kjorlien (2001) 
states, hiring part-time faculty should be an intentional decision and not one that has to be 
undertaken as a temporary solution to a university’s recruiting problems. Yet many schools are 
falling back to part-time, temporary instructors to fill the gap. In many cases, hiring criteria 
cannot be met but it also appears that they cannot be changed so positions remain open. 

For teaching institutions that often utilize large numbers of adjunct faculty, initial screening 
criteria are often different. Whereas many believe that AACSB accredited schools and research 
institutions should, at a minimum, require doctoral degrees even for adjuncts (Sixl-Danill, 
Willians, & Wong, 2006), many teaching institutions place more emphasis on practice and tend 
to hire master’s level instructors with significant industry experience. However, a focus on 
practice does not exempt teaching institutions from pressures, or even requirements in some 
cases, to build up the number of doctorates they have on staff and the pressure continues to 
mount for these institutions to comply with these expectations. 

Additionally, at research institutions, faculty must often meet other measures of performance in 
order to be qualified for hire or tenure. Often cited are evidence of empirical scholarship, records 
of good teaching, and, in some cases, a history of successful grant applications (Arreola, 2000). 
In an adult learner focused institution that uses a practitioner-faculty model, many of these 
factors are not stressed to the same degree. The only clear overlap between the two is quality 
teaching. 

To what extent do varying qualifications and performance metrics present a problem for either 
type of university? Are the research institutions spending too much time competing for a 
shrinking number of available doctorates when there is much to gain from practical experience? 
Should the minimum requirement for every faculty member, including adjuncts, be a doctorate? 
Are the teaching schools cutting comers by not requiring similar levels of education, teaching 
experience, and scholarship from their faculty? Should such institutions give in to the pressures 
to hire more doctorates with evidence of scholarship? If so, where is the evidence to show that 
such qualities are linked to quality teaching performance in faculty members at adult-focused 
teaching institutions? 
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Evaluating Instruction and Performance 

Tracking the performance of faculty members has raised a number of concerns largely due to the 
fact that one of the primary indicators of faculty performance has been student evaluations. Such 
evaluations are commonly the subject of debate regarding their use, validity, and reliability 
(Theall, Abrami, & Mets, 2001). Other concerns with the use of student evaluations center on 
bias introduced by low response rates and non-responder bias, which are particularly problematic 
when the evaluations are conducted online (Thorpe, 2002). Another issue cited with student 
evaluations is that they are frequently found to be positively correlated with grade expectations 
(the higher the grade the student expects to receive, the better the evaluation). Additionally, 
when student evaluations are mandatory, and all students are required to participate, evidence 
has been found to suggest a link to grade inflation (Soper, 1973). 

Yet, despite these problems, certain steps can be taken to increase the usefulness of the data that 
come from student evaluations. According to Franklin (2005), most researchers agree that 
student evaluations are highly reliable and that most students tend to agree with each other in 
their ratings of an instructor. Additionally, Centra (1993) indicated that most researchers agree 
that student evaluations are at least moderately valid, and that student ratings correlate positively 
with other measures of teaching effectiveness. 

Other problems with student evaluations stemming from low response rates may not be 
warranted. Thorpe (2002) found no evident pattern to suggest that web-based responses would be 
significantly different from responses offered in class even if the response rates were variable. 
Lastly, many concerns have been raised regarding the subjectivity of student evaluations. This 
has been shown to be a problem that can have a substantial impact on the validity of the data 
collected. However, steps can be taken to minimize this impact as well. Arreola (2000) suggests 
that student evaluations can be structured to focus on specific, objective actions that can be 
observed. Students can be asked to rate instructors on their tendency to come to class on time or 
provide timely feedback, as opposed to asking the students how they would rate the overall 
quality of an instructor. 

Thus, it would appear that many researchers believe that with proper cautions, student 
evaluations can be an effective measure of faculty performance, if steps are taken to both correct 
for systematic bias and limit the subjectivity of the evaluation items. According to Marsh (as 
cited in Theall, Abrami, & Mets, 2005) “There is a wealth of evidence that when ratings 
instruments are properly constructed, properly administered, properly analyzed, properly 
reported, and used in a comprehensive systematic way, they can provide information felt by 
teachers, administrators and students to be useful” (p. 1). 

The question remains whether student evaluation results can be used to provide support to the 
consideration of different hiring qualifications that should be sought in faculty candidates. Can a 
link be found between the factors used for an initial candidate screening and instructor 
performance as measured by student evaluations or other measure of performance to suggest a 
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desirable candidate profile for a teaching institution? Are there any clear indicators of candidate 
characteristics that hiring managers at teaching institutions should be screening for? The study 
described in the sections to follow sheds light on these questions. 

The University 

City University of Seattle, headquartered in Bellevue, WA, is primarily a degree completion and 
graduate university that focuses on an adult student population. Since its founding in 1973, the 
university has valued practical education and has utilized a network of practitioner faculty who 
are hired for their industry expertise. Courses are designed to allow the faculty to teach skills that 
are directly applicable to the careers and lives of their students. 

Because of its reliance on a large number of adjunct faculty, the university continually hires new 
faculty members who need to be trained, oriented, and managed. This becomes a complex task 
given that the university employs over 700 adjunct faculty members across its three schools. 

With 252 adjunct instructors, the university’s School of Management encounters unique 
challenges as the majority of its faculty are hired for their subject matter expertise and typically 
do not have formal training in adult learning techniques. On average, the School of Management 
hires about 100 new faculty members per year and, once hired, these faculty need additional 
oversight and training to be effective as, often, subject matter expertise is necessary but 
insufficient to ensure effective instruction (Arreola, 2000). 

Any methods that can be identified to streamline the faculty screening, selection, hiring, and 
training processes could have an important impact on the amount of time that needs to be 
dedicated to each faculty hire. This would be especially true if the School of Management was 
able to identify faculty candidates who are likely to be successful instructors, based on attributes 
that could be identified during an initial screening of the candidate’s CV. Such attributes would 
include aspects of the applicant’s education, experience, or scholarship. Having a profile for a 
“highly qualified” applicant, where the qualifications were actually linked to success in the 
classroom, could be of great assistance to the hiring supervisors in the School of Management. 

Identification of Highly Qualified Faculty 

Recently, the university engaged in a faculty profile project to gather information about its 
adjunct practitioner faculty. A web-based survey form was created that allowed respondents to 
provide information about their education, experience, and scholarly achievements. It was hoped 
that once the data were collected they could be used to identify whether any of the factors in the 
faculty profiles could be used as potential indictors of likely faculty performance. This could 
potentially help refine the typical screening profile of a successful faculty candidate for the 
School of Management. It was believed that once the data were collected for all of the current 
faculty, that certain attributes would emerge in consistent patterns and that these attributes could 
be used as part of the screening criteria for new faculty hires. 
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During this effort, the university examined a group of top-performing faculty in the School of 
Management and compared their characteristics to a group of faculty with moderate performance 
metrics to see if 1) there was any difference in selected faculty profile factors and 2) if there were 
any factors that correlated with performance. The sections below describe the results of this 
effort. 



Methods 



End of Course Evaluations 

The university employs an end of course evaluation (EOCE) tool consisting of 16 instructor- 
related and 6 course-related statements. Students are asked to respond using a Likert scale of 1 - 
5, with 5 representing the strongest agreement with each statement about the instructor’s 
behavior. Students provide feedback on several different aspects of the instructor’s performance 
and the quality of the course. Both the instructor and course sections of the evaluation are 
summarized by calculating a simple average of the responses to the questions in that section. The 
average of the instructor-related questions is used as a metric to gauge overall student 
satisfaction with instructor performance, yet, at any time, data can be easily pulled for all 16 
instructor-related characteristics. A copy of the university’s EOCE tool can be found in the 
appendix. 

Many of the instructor-related questions are based on the work of Chickering and Gamson 
(1987) and are intended to link back to their seven practices of effective instruction. Other 
questions are founded in adult learning theory and link back to the tenets of Knowles, Holton & 
Swanson (1998). In the initial design, efforts were made to limit the subjectivity of the 
evaluation questions by focusing on specific instructor behaviors, yet subjectivity is a fact of 
student evaluations and the results reported in this study are impacted by an amount of 
unavoidable subjectivity. 

For the purpose of this study, cumulative EOCE data were gathered on a population of 
instructors from spring of 2009 to as far back as 2006, which is when the university began 
collecting EOCE responses in a web-based format. Some veteran instructors had taught many 
sections that spanned the past four academic years and, as a result, had a large pool from which 
to pull EOCE data, whereas others may have had data for just the past academic year. These data 
were collected to be used as one metric of faculty performance to be compared to the 
characteristics that faculty reported in their profiles. Three values were recorded for each faculty 
member: EOCE instructor score average for the entire period, number of possible responses, and 
the number of actual responses. From the last two figures, a response rate was calculated. 

The EOCE response data also served as a filter for the faculty members who were included in 
this study. This was to eliminate any faculty for whom there were no data or too little data to be 
of any use. The filter eliminated any faculty members who had not taught at least five sections 
from which EOCE data could be collected, or did not return at least 10% of the possible EOCE 
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responses for all sections. Another filter eliminated faculty members who had not updated their 
faculty profiles. Once these filters were applied, the study’s sample was reduced to 101 
instructors. Most of the eliminated faculty members were new instructors who had not yet taught 
five full sections. 

Classifying the Instructors 

Once the EOCE data were gathered, the faculty members were ranked and then classified into 
four quartiles based on their average instructor scores. The purpose of this classification was to 
enable comparisons of faculty profile characteristics across groups of faculty with different 
EOCE averages. The average instructor scores for these four quartiles were: 

Table 1 

Average instructor EOCE scores by quartile 



Quartile 


Average 


One 


4.55 


Two 


4.32 


Three 


4.15 


Four 


3.86 


n for quartile 1-3 = 25 




n for quartile 4 = 26 





Top-Performing Faculty 

The next step was to gather a group of faculty members who were identified as top performers 
by their primary supervisors. This was done to have a second group of highly rated instructors 
that were identified in a way that did not involve EOCEs. That way, there were two measures of 
effectiveness to compare against the weakest group. 

The identification of these faculty members was done by asking each Primary Supervisor in the 
School of Management to name his or her top instructors based on factors such as instructional 
quality, job performance, service to students, and subject matter expertise. This yielded a list of 
25 faculty members. This top performing group had an average instructor score of 4.32 on the 
EOCEs. 

When comparing this list to the original list of 101 faculty members identified for this study, it 
was discovered that the top-rated faculty were fairly widely distributed among the four quartiles 
with most clustering into quartile one. There were, however, notable intersections between each 
of the quartiles and the list of top performers. These intersections are as follows: 
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Table 2 

Top performers intersected by quartile 


Quartile 


Intersection with Top 
Performers 


One 


10 


Two 


4 


Three 


6 


Four 


5 



Faculty Profiles 

Next, data were pulled from the faculty profiles for each faculty member who was in the top 
quartile, the bottom quartile, and those who had made the list of top performers. The factors for 
which data were gathered included: 

• Graduate degrees 

• Years of professional experience 

• Years of instructional experience external to the university 

• Years of instructional experience within the university 

• Scholarly publications 

• Scholarly presentations 

• Other scholarly activities 

The data from each of these factors was compared across instructor groups to determine if the 
following questions could be answered: 

1. Were there any significant differences between cumulative or average values for the 
profile factors when comparing the faculty members in quartile one and quartile four? 

2. Were there any significant differences when comparing the top-rated faculty and quartile 
four? 

3. Were there any correlations between the profile factors and the scores on the EOCEs for 
faculty in quartile four when combined with the top-rated performers? 

4. Can any of the factors be used as potential screening indicators to help predict highly 
successful faculty? 



Results 



Comparisons 

Table 3, below, shows a breakdown of the findings from the faculty profiles that were analyzed 
in this study. The main comparisons were made between quartile one versus quartile four and the 
list of top performers also compared against quartile four. The second comparison required 
adjustment due to the intersection of the five faculty members who were included in both groups. 
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Table 3 



Cross group comparison of faculty factors 



Factors 


Toprated 


Quartile 

One 


Adjusted 

Quartile 

Four 


Average EOCE Scores 


4.32 


4.55 


3.88 


Average Possible EOCE Responses 


315.3 


220.5 


211.6 


Average Actual EOCE Responses 


127.2 


94.3 


72.6 


Average EOCE Response Rate 


37.8 


43.1 


35.8 


Number of Doctoral Degrees (Including JDs) 


8 


8 


6 


Number of Master's Degrees 


25 


27 


31 


Number of Certifications 


18 


23 


18 


Average Years of Professional Experience 


23.3 


19.9 


24.5 


Average Years of Instructional Experience - 
External 


5.4 


6.6 


6 


Average Years of Instructional Experience - 
Internal 


8.8 


7.5 


6.5 


Scholarly Publications 


15 


28 


30 


Scholarly Presentations 


20 


17 


25 


Other Scholarly Work Including Consulting and 
Instructional Design 


4 


8 


6 



To ensure mutual exclusivity between the groups, the five top-rated faculty members were 
removed from the fourth quartile and replaced with the five faculty members from quartile three 
that had the lowest EOCE scores. Both the top-rated group and quartile one had 25 members; the 
fourth quartile had 26 members. 

The top-rated group had a significantly higher number of possible EOCE responses indicating 
that they teach more classes. This makes sense given that these faculty members were identified 
by their primary supervisors who are usually the ones who select them to teach classes. There 
was a noticeably higher EOCE response rate in quartile one. This is also the group that had the 
highest EOCE average score. This may suggest that among the stronger faculty, higher EOCE 
response rates may lead to better scores. 

The initial findings about educational levels are perhaps the most surprising given the emphasis 
on the importance of doctoral degrees in academia. The two top-rated groups each had a total of 
eight doctorates among them whereas the lower group had only six. These differences do not rise 
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to the level of significance primarily due to the low sample size. Perhaps if the numbers of 
faculty being compared were higher, a significant difference could be found. 

When comparing the number of master’s degrees in each of the groups, the numbers are higher 
(some faculty members held multiple master’s degrees) but there still does not seem to be any 
significant differences between the three groups. The faculty in quartile four had six more 
master’s degrees that the top performers and four more than the faculty in quartile one. 
Essentially, the metrics tracked for education yielded no useful data to differentiate one group 
from another. 

Some interesting results were also found regarding professional experience, which is important 
given the emphasis on practical experience at the university. The faculty in quartile four had the 
most professional experience, whereas the faculty in quartile one had the least. Similar results 
were reported regarding academic experience. There did not seem to be any differences in 
experience levels outside of the university, however, top-rated instructors had the highest 
average years of experience teaching at the university, which, again, is not surprising given that 
the university makes efforts to keep and use its strongest instructors. 

Overall, average years of experience, both professional and academic, was fairly static across the 
three groups except for a notable drop in the average years of professional experience in quartile 
one which is different from the same figure for the top performers and leads to no useful 
conclusion in regards to the main objectives of this study. There were also no significant 
differences in the number of scholarly publications, presentations, or other scholarly work except 
for what looks to be an anomalous drop in the number of scholarly publications for the top-rated 
group. 

Correlations 

As a further measure, correlations were calculated to determine the extent to which some of the 
quantifiable factors were matched with EOCE scores. This step was taken to further highlight the 
relationships between the EOCEs and the faculty factors and to ensure that no outliers were 
adversely impacting the comparisons shown in table 3. To calculate the correlations, the list of 
faculty members in the top performer group was combined with the list of faculty members in 
the adjusted fourth quartile and correlation coefficients were calculated by comparing EOCE 
average scores to EOCE metrics, years of professional experience, and years of instructional 
experience (inside the university, outside the university, and combined). 

The correlation coefficients can be found in table 4. There was a minor, yet probably 
meaningless, correlation between both possible / actual EOCE responses and the EOCE average 
scores. However, there does not appear to be any significant correlation between EOCE average 
scores and response rates. There also does not appear to be any real correlation between the 
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number of years of professional experience and EOCE scores. The same holds true for the 
comparison to the number of years of educational experience outside of the university. There is 



Table 4 

Correlations between EOCEs and Faculty Factors 


EOCE Averages Correlated With: 


r 


EOCE Possible Responses 


0.27 


EOCE Actual Responses 


0.23 


EOCE Response Rate 


0.15 


Years Professional Experience 


0.10 


Years External Educational Experience 


0.09 


Years Internal Educational Experience 


0.30 


Combined Educational Experience 


0.28 



a small, positive correlation between EOCE scores and the number of years experience within 
the university but this can be attributed mostly to the reliance on qualified faculty and the 
university’s tendency to eliminate faculty who do not perform to expectation. 

Limitations 

The data gathered for this project and the conclusions that can be drawn from them are subject to 
some clear limitations, the biggest of which is the fact that the primary measure of faculty 
performance was student evaluations, which are subject to a number of issues often having to do 
with validity. It is clear that, in general, student evaluations are intended to measure student 
perceptions of faculty performance and this study is no different. However, as Centra (1993) 
pointed out, student evaluations often correlate with other measures of faculty performance. 
Furthermore, efforts were taken to ensure that the student evaluations in this study focused on 
objective measures of faculty performance as Arreola (2000) recommended. Thus, for the most 
part, there is reason to believe that the use of student evaluations, as used in this study, can be a 
useful measure of faculty performance. 

A second limitation is that the data regarding the faculty factors were gathered on instructors 
who had been teaching at the university long enough to have taught at least five sections. Some 
faculty members were in their first year whereas others had been teaching at the university for 
more than ten years. The goal of this study was to see if there were factors that were evident at 
the time of hire that might correlate to strong faculty performance yet the faculty members who 
were measured were not new. This is a valid concern given that the strongest correlation found in 
this study was between EOCE scores and the number of years that the faculty member had been 
teaching for the university. Yet, in some ways, this limitation actually strengthens this study in 
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that it allows the focus to be only on faculty who have met a certain performance threshold and it 
eliminates those instructors whose performance was not satisfactory to be retained long enough 
to teach five sections. Still, it could be argued that there may be value in expanding this study to 
include faculty members who did not meet performance thresholds, something that would not 
have been possible in this study since the university did not gather faculty profiles on these 
instructors. 



Conclusion 

The initial conclusion is that there does not appear to be a useful difference in the measures of 
the quantifiable factors in the faculty profiles among the three groups identified for this study. 
When the groups are adjusted to remove overlaps, the results do not improve. Neither education 
level, professional or academic experience, nor scholarly activity proved to be a clear indicator 
of instructor success. Given this, it does not appear that the university’s School of Management 
will benefit much by creating a “highly qualified” faculty profile to assist with faculty candidate 
screening, at least not one that is based on the factors examined in this study. Thus, the School of 
Management should now turn its attention toward other means of applicant screening such as 
carefully structured interview questions or teaching demonstrations, both of which could be 
strengthened through a validation process similar to the one that was attempted in this study. 

This could be a significant area of future research for business programs in teaching universities. 
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Appendix 

End of Course Evaluation Questions 

Students are asked to respond using a Likert scale of 1 - 5, with 5 representing the strongest 
agreement with each statement. 

Instructor Statements 

1. The instructor encourages students to participate actively in the course. 

2. The instructor provides opportunities for students to learn from each other. 

3. The instructor communicates the value of the course. 

4. The instructor responds to my questions in a timely fashion. 

5. The instructor facilitates a supportive learning environment. 

6. The instructor communicates the course content in a clear and organized way. 

7. The instructor clearly explains the course learning goals, assignments, and grading 
criteria. 

8. The instructor uses class time effectively. 

9. The instructor is knowledgeable about the subject matter. 

10. The instructor effectively uses a variety of teaching strategies. 

11. The instructor connects the course to real-life situations. 

12. The instructor provides useful feedback on my progress in the course. 

13. The instructor grades assignments in alignment with the grading criteria. 

14. The instructor respects diverse styles and ways of learning. 

15. The instructor asks for periodic feedback from students to improve his or her teaching. 

16. Overall, the instructor is very effective in teaching this course. 

A section for comments about the instructor is provided at the end of the instructor section. 
Course Statements 

17. The course connects learning to real-world situations. 

18. The learning goals for this course are clear. 

19. The projects and assignments help me meet the learning goals of the course. 

20. The required resources contribute to my learning in this course. 

21. The work load is logically distributed throughout the course. 

22. Overall, the way this course is designed is effective in helping me achieve the learning 
goals. 

A section for comments about the course is provided at the end of the course section. 
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